Providing information associated with an identified representation of an object

ABSTRACT

Methods, apparatus systems and computer program products are described herein that provide for using video or still shot analysis, such as AR or the like, to assist the user of mobile devices with receiving information corresponding to an abstraction or representation of a subject. Some subjects are difficult to capture in a video or still shot. The method and devices described herein capture representations of difficult to capture or unavailable subjects and presents information related to the subject with the representation. In an embodiment, the representation is a screenshot and the information is provided related to the application that is represented by the screenshot. Various other types of representations including depictions, advertisements, portions of, and identifying marks can be identified by the system and method and information presented relating to the corresponding subjects. In some cases, the information is customized with financial information of the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 61/450,213, filed Mar. 8, 2011, entitled “Real-Time Video ImageAnalysis Applications for Commerce Activity,” and U.S. ProvisionalPatent Application Ser. No. 61/478,407, filed Apr. 22, 2011, entitled“Providing Information Associated with an Identified Representation ofan Object,” the entirety of each of which is incorporated herein byreference.

BACKGROUND

Modern handheld mobile devices, such as smart phones or the like,combine multiple technologies to provide the user with a vast array ofcapabilities. For example, many smart phones are equipped withsignificant processing power, sophisticated multi-tasking operatingsystems, and high-bandwidth Internet connection capabilities. Moreover,such devices often have addition features that are becoming increasingmore common and standardized features. Such features include, but arenot limited to, location-determining devices, such as Global PositioningSystem (GPS) devices; sensor devices, such as accelerometers; andhigh-resolution video cameras.

As the hardware capabilities of such mobile devices have increased, sotoo have the applications (i.e., software) that rely on the hardwareadvances. One such example of innovative software is a category known asaugmented reality (AR), or more generally referred to as mediatedreality. One such example of an AR application platform is Layar,available from Layar, Amsterdam, the Netherlands.

The Layar platform technology analyzes location data, compass directiondata, and the like in combination with information related to theobjects, locations or the like in the video stream or still shot tocreate browse-able “hot-spots” or “tags” that are superimposed on themobile device display, resulting in an experience described as “realitybrowsing”.

In some situations, objects are abstract and cannot be adequatelycaptured in a video stream or still shot. For example, software programssuch as mobile device applications are abstract and difficult to capturein a video stream or still shot. Further, some objects are concrete butcommonly reproduced in abstract form. For example, photographs ofindividuals are an abstraction of the person represented therein. Also,some objects are represented by abstract symbols or identifiers, such astrademarks or logos. Individuals are interested in the objectsrepresented by these abstractions, not in the abstractions themselves.However, individuals may find it difficult to gather information fromabstractions because they are not the actual object of interest but onlyrepresent the object in some form.

Therefore, a need exists to implement video or still shot analysis,i.e., AR or the like, to assist the user of mobile devices withreceiving information based on an abstraction of an object.

SUMMARY

The following presents a simplified summary of one or more embodimentsin order to provide a basic understanding of such embodiments. Thissummary is not an extensive overview of all contemplated embodiments,and is intended to neither identify key or critical elements of allembodiments nor delineate the scope of any or all embodiments. Its solepurpose is to present some concepts of one or more embodiments in asimplified form as a prelude to the more detailed description that ispresented later.

Methods, apparatus systems and computer program products are describedherein that provide for using video or still shot analysis, such as ARor the like, to assist the user of mobile devices with receivinginformation corresponding to an abstraction or representation of asubject. Through the use of real-time video, objects, logos, artwork,products, locations and other features that can be recognized in thereal-time video stream can be matched to data associated with such toassist the user with receiving information based on representationsrelated to subjects that are not easily capturable in video or stillshots. In specific embodiments, the data that is matched to the imagesin the real-time video stream is specific to financial institutions,such as customer financial behavior history, customer purchasepower/transaction history and the like. In this regard, many of theembodiments herein disclosed leverage financial institution data, whichis uniquely specific to a financial institution, in providinginformation to mobile device users in connection with real-time videostream analysis.

To the accomplishment of the foregoing and related ends, the one or moreembodiments comprise the features hereinafter fully described andparticularly pointed out in the claims. The following description andthe annexed drawings set forth in detail certain illustrative featuresof the one or more embodiments. These features are indicative, however,of but a few of the various ways in which the principles of variousembodiments may be employed, and this description is intended to includeall such embodiments and their equivalents.

In some embodiments, a method of presenting information based on acaptured representation of a subject is provided. In an embodiment, themethod includes recognizing, via a mobile device, the representation ofthe subject; identifying, via a computing device processor, the subjectbased on an analysis of the representation; retrieving informationcorresponding to the subject; and presenting information using themobile device, the information related to the subject presented inconjunction with presentation of the representation.

In some embodiments, the method also includes capturing therepresentation from a still shot and analyzing the still shot toidentify the subject. Optionally, the method can include capturing therepresentation from a video and analyzing the video to identify thesubject. In some embodiments, the still shot or video are captured inreal time while in other embodiments the still shot or video or saved asa file and provided to the mobile device for analysis. In someembodiments, the method customizes the information with financialinformation of a user of the mobile device or of the subject. Forexample, the information can be customized with account information,purchase history, and/or demographic data of the user.

The representation disclosed herein can vary widely. In an exemplaryembodiment, the representation is a screenshot of an applicationinstalled on or executing on a computing device. The computing devicemay be any type of personal or business computing device, such as amobile phone, a desktop computer, a laptop computer, a tablet computer,a video game console, or a personal digital assistant. In otherembodiments, the representation is a depiction, advertisement, portionof, or identifying mark for a subject.

Embodiments of the invention also provide an apparatus where theapparatus has a device for capturing a video stream or still shot, aprocessor configured to determine which representations from the videostream or still shot are associated with subjects, and a display forpresenting the video stream with information, the information beingassociated with the subjects corresponding to the representations. Insome embodiments, the information is presented real-time in the videostream. In further embodiments, the apparatus includes a microphone andthe subject is identified, at least in part by an audio recording fromthe microphone.

The apparatus also includes a computing platform having a processor, amemory in communication with the processor, and logic stored in thememory and executable by the processor. In some embodiments, the logicincludes image capture logic configured to capture a video streamcomprising a representation of the subject. The logic may also includeanalysis logic configured to analyze the representation of the subject,image identification logic configured to identify the subject from theanalysis of the representation, and presentation logic configured topresent information related to the subject in the video.

In some embodiments, the apparatus also includes a network interfacedevice configured to wirelessly connect to a database comprising aplurality of subjects and associated representations. In still furtherembodiments, the apparatus includes customization logic stored in thememory and executable by the processor, wherein the customization logicis configured to customize a presentation for the user. The presentationcan be customized with the user's financial information, such asprevious transactions between the user and the identified subject.

Embodiments of the invention also provide a computer program productcomprising a non-transitory computer-readable medium havingcomputer-executable code stored thereon. In one embodiment, thecomputer-executable code includes: a first code portion configured tocapture, via a mobile device, a video stream or still picture, a secondcode portion configured to determine which representations from thevideo stream or still shot are associated with subjects, and a thirdcode portion configured to present on a display the video stream orstill shot information on the subjects corresponding to therepresentations. In some embodiments, the computer program product alsoincludes a database comprising a plurality of subjects and relatedrepresentations. The computer program product may be stored on a mobiledevice such as a mobile phone, a personal digital assistant, a camera,or a tablet computer. In further embodiments, the computer programproduct includes instructions for customizing the information for theuser, such as by providing information related to at least one previoustransaction of the user or by providing coupons for the user.

The features, functions, and advantages that have been discussed may beachieved independently in various embodiments of the present inventionor may be combined with yet other embodiments, further details of whichcan be seen with reference to the following description and drawings

BRIEF DESCRIPTION OF THE DRAWINGS

Having thus described embodiments of the invention in general terms,reference will now be made to the accompanying drawings, which are notnecessarily drawn to scale, and wherein:

FIG. 1 is a block diagram illustrating a mobile device, in accordancewith an embodiment of the invention;

FIG. 2 is a block diagram illustrating an AR environment, in accordancewith an embodiment of the invention;

FIG. 3 is a block diagram illustrating a mobile device, in accordancewith an embodiment of the invention;

FIG. 4 is a high level flow chart of a method for presenting informationbased on a captured representation of a subject, in accordance with anembodiment of the invention;

FIG. 5 is a block diagram illustrating an AR environment having bothrepresentations and subjects, in accordance with an embodiment of theinvention;

FIG. 6 is a block diagram of the method for presenting informationwherein the representation is a screenshot of an application, inaccordance with an embodiment of the invention;

FIG. 7 is a block diagram of the method for presenting informationwherein the representation is a depiction of a subject, in accordancewith an embodiment of the invention;

FIG. 8 is a block diagram of the method for presenting informationwherein the representation is an advertisement for a subject, inaccordance with an embodiment of the invention;

FIG. 9 is a block diagram of the method for presenting informationwherein the representation is a portion of a subject, in accordance withan embodiment of the invention; and

FIG. 10 is a block diagram of the method for presenting informationwherein the representation is an identifying mark, in accordance with anembodiment of the invention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Embodiments of the present invention will now be described more fullyhereinafter with reference to the accompanying drawings, in which some,but not all, embodiments of the invention are shown. Indeed, theinvention may be embodied in many different forms and should not beconstrued as limited to the embodiments set forth herein; rather, theseembodiments are provided so that this disclosure will satisfy applicablelegal requirements. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea thorough understanding of one or more embodiments. It may be evident;however, that such embodiment(s) may be practiced without these specificdetails. Like numbers refer to like elements throughout.

Various embodiments or features will be presented in terms of systemsthat may include a number of devices, components, modules, and the like.It is to be understood and appreciated that the various systems mayinclude additional devices, components, modules, etc. and/or may notinclude all of the devices, components, modules etc. discussed inconnection with the figures. A combination of these approaches may alsobe used.

The steps and/or actions of a method or algorithm described inconnection with the embodiments disclosed herein may be embodieddirectly in hardware, in a software module executed by a processor, orin a combination of the two. A software module may reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a harddisk, a removable disk, a CD-ROM, or any other form of storage mediumknown in the art. An exemplary storage medium may be coupled to theprocessor, such that the processor can read information from, and writeinformation to, the storage medium. In the alternative, the storagemedium may be integral to the processor. Further, in some embodiments,the processor and the storage medium may reside in an ApplicationSpecific Integrated Circuit (ASIC). In the alternative, the processorand the storage medium may reside as discrete components in a computingdevice. Additionally, in some embodiments, the events and/or actions ofa method or algorithm may reside as one or any combination or set ofcodes and/or instructions on a machine-readable medium and/orcomputer-readable medium, which may be incorporated into a computerprogram product.

In one or more embodiments, the functions described may be implementedin hardware, software, firmware, or any combination thereof. Ifimplemented in software, the functions may be stored or transmitted asone or more instructions or code on a computer-readable medium.Computer-readable media includes both computer storage media andcommunication media including any medium that facilitates transfer of acomputer program from one place to another. A storage medium may be anyavailable media that can be accessed by a computer. By way of example,and not limitation, such computer-readable media can comprise RAM, ROM,EEPROM, CD-ROM or other optical disk storage, magnetic disk storage orother magnetic storage devices, or any other medium that can be used tocarry or store desired program code in the form of instructions or datastructures, and that can be accessed by a computer. Also, any connectionmay be termed a computer-readable medium. For example, if software istransmitted from a website, server, or other remote source using acoaxial cable, fiber optic cable, twisted pair, digital subscriber line(DSL), or wireless technologies such as infrared, radio, and microwave,then the coaxial cable, fiber optic cable, twisted pair, DSL, orwireless technologies such as infrared, radio, and microwave areincluded in the definition of medium. “Disk” and “disc”, as used herein,include compact disc (CD), laser disc, optical disc, digital versatiledisc (DVD), floppy disk and blu-ray disc where disks usually reproducedata magnetically, while discs usually reproduce data optically withlasers. Combinations of the above should also be included within thescope of computer-readable media.

Thus, methods, systems, computer programs and the like are hereindisclosed that provide for using real-time video analysis, such as AR orthe like to assist the user of mobile devices with receiving informationrelated to subjects corresponding to abstract representations in anenvironment. Through the use of real-time object recognition, objects,logos, artwork, products, locations and other features that can berecognized in the real-time video stream can be matched to dataassociated with such to assist the user with purchasing and/orcustomizing subjects represented in the AR environment. In specificembodiments, the data that is matched to the images in the real-timevideo stream is specific to financial institutions, such as customerfinancial behavior history, customer purchase power/transaction historyand the like. In this regard, many of the embodiments herein disclosedleverage financial institution data, which is uniquely specific tofinancial institution, in providing information to mobile devices usersin connection with real-time video stream analysis.

While embodiments discussed herein are generally described with respectto “real-time video streams” or “real-time video” it will be appreciatedthat the video stream may be captured and stored for later viewing andanalysis. Indeed, in some embodiments video is recorded and stored on amobile device and portions or the entirety of the video may be analyzedat a later time. The later analysis may be conducted on the mobiledevice or loaded onto a different device for analysis. The portions ofthe video that may be stored and analyzed may range from a single frameof video (e.g., a screenshot) to the entirety of the video.Additionally, rather than video, the user may opt to take a stillpicture of the environment to be analyzed immediately or at a latertime. Embodiments in which real-time video, recorded video or stillpictures are analyzed are contemplated herein.

FIG. 1 illustrates an embodiment of a mobile device 100 that may beconfigured to execute object recognition and augmented reality (AR)functionality, in accordance with specific embodiments of the presentinvention. A “mobile device” 100 may be any mobile communication device,such as a cellular telecommunications device (i.e., a cell phone ormobile phone), personal digital assistant (PDA), a mobile Internetaccessing device, or other mobile device including, but not limited toportable digital assistants (PDAs), pagers, mobile televisions, gamingdevices, laptop computers, cameras, video recorders, audio/video player,radio, GPS devices, any combination of the aforementioned, or the like.

The mobile device 100 may generally include a processor 110 communicablycoupled to such devices as a memory 120, user output devices 136, userinput devices 140, a network interface 160, a power source 115, a clockor other timer 150, an image capture device 170 such as a camera, apositioning system device 175 (e.g., a Global Positioning System (GPS)device), one or more integrated circuits 180, etc.

In some embodiments, the mobile device 100 and/or the server access oneor more databases or datastores (not shown in FIG. 1) to search forand/or retrieve information related to the object and/or marker. In someembodiments, the mobile device 100 and/or the server access one or moredatastores local to the mobile device 100 and/or server and in otherembodiments, the mobile device and/or server access datastores remote tothe mobile device 100 and/or server. In some embodiments, the mobiledevice 100 and/or server access both a memory and/or datastore local tothe mobile device and/or server as well as a datastore remote from themobile device and/or server

The processor 110, and other processors described herein, may generallyinclude circuitry for implementing communication and/or logic functionsof the mobile device 100. For example, the processor 110 may include adigital signal processor device, a microprocessor device, and variousanalog to digital converters, digital to analog converters, and/or othersupport circuits. Control and signal processing functions of the mobiledevice 100 may be allocated between these devices according to theirrespective capabilities. The processor 110 thus may also include thefunctionality to encode and interleave messages and data prior tomodulation and transmission. The processor 110 may additionally includean internal data modem. Further, the processor 110 may includefunctionality to operate one or more software programs or applications,which may be stored in the memory 120. For example, the processor 110may be capable of operating a connectivity program, such as a webbrowser application 122. The web browser application 122 may then allowthe mobile device 100 to transmit and receive web content, such as, forexample, location-based content and/or other web page content, accordingto a Wireless Application Protocol (WAP), Hypertext Transfer Protocol(HTTP), and/or the like.

The processor 110 may also be capable of operating applications, such asan object recognition application 125. The object recognitionapplication 125 may be downloaded from a server and stored in the memory120 of the device 100. Alternatively, the object recognition application125 may be pre-installed and stored in a memory in the integratedcircuit 180. In such an embodiment, the user may not download the objectrecognition application 125 from a server.

In some embodiments, the processor 110 may also be capable of operatingone or more applications, such as one or more applications functioningas an artificial intelligence (“AI”) engine. The processor 110 mayrecognize objects that it has identified in prior uses by way of the AIengine. In this way, the processor 110 may recognize specific objectsand/or classes of objects, and store information related to therecognized objects in one or more memories and/or databases discussedherein. Once the AI engine has thereby “learned” of an object and/orclass of objects, the AI engine may run concurrently with and/orcollaborate with other modules or applications described herein toperform the various steps of the methods discussed. For example, in someembodiments, the AI engine recognizes an object that has been recognizedbefore and stored by the AI engine. The AI engine may then communicateto another application or module of the mobile device and/or server, anindication that the object may be the same object previously recognized.In this regard, the AI engine may provide a baseline or starting pointfrom which to determine the nature of the object. In other embodiments,the AI engine's recognition of an object is accepted as the finalrecognition of the object.

The processor 110 may also be capable of operating applications, such asan AR presentation application 121. The AR presentation application 121may be downloaded from a server and stored in the memory 120 of themobile device 100. Alternatively, the AR presentation application 121may be pre-installed and stored in a memory in the Integrated circuit180. In such an embodiment, the user may not download the ARpresentation application 121 from a server.

The integrated circuit 180 may include the necessary circuitry toprovide the object recognition and/or augmented reality functionality tothe mobile device 100. Generally, the integrated circuit 180 willinclude data storage 171 which may include data associated with theobjects within a video stream that the object recognition application125 identifies as having a certain marker(s). The integrated circuit 180and/or data storage 171 may be an integrated circuit, a microprocessor,a system-on-a-chip, a microcontroller, or the like. As discussed above,in one embodiment, the integrated circuit 180 may provide the objectrecognition and/or the AR functionality to the mobile device 100.

Of note, while FIG. 1 illustrates the integrated circuit 180 as aseparate and distinct element within the mobile device 100, it will beapparent to those skilled in the art that the integrated circuitfunctionality may be incorporated within other elements in the mobiledevice 100. For instance, the functionality of the integrated circuit180 may be incorporated within the mobile device memory 120 and/orprocessor 110. In a particular embodiment, the functionality of theintegrated circuit 180 is incorporated in an element within the mobiledevice 100 that provides object recognition and/or AR capabilities tothe mobile device 100. In some embodiments, the integrated circuit 180functionality is present in the firmware of the mobile device 100. Stillfurther, the integrated circuit 180 functionality may be included in aremovable storage device such as an SD card or the like.

The processor 110 may be configured to use the network interface 160 tocommunicate with one or more other devices on a network. In this regard,the network interface 160 may include an antenna 176 operatively coupledto a transmitter 174 and a receiver 172 (together a “transceiver”). Theprocessor 110 may be configured to provide signals to and receivesignals from the transmitter 174 and receiver 172, respectively. Thesignals may include signaling information in accordance with the airinterface standard of the applicable cellular system of the wirelesstelephone network that may be part of the network. In this regard, themobile device 100 may be configured to operate with one or more airinterface standards, communication protocols, modulation types, andaccess types. By way of illustration, the mobile device 100 may beconfigured to operate in accordance with any of a number of first,second, third, and/or fourth-generation communication protocols and/orthe like. For example, the mobile device 100 may be configured tooperate in accordance with second-generation (2G) wireless communicationprotocols IS-136 (time division multiple access (TDMA)), GSM (globalsystem for mobile communication), and/or IS-95 (code division multipleaccess (CDMA)), or with third-generation (3G) wireless communicationprotocols, such as Universal Mobile Telecommunications System (UMTS),CDMA2000, wideband CDMA (WCDMA) and/or time division-synchronous CDMA(TD-SCDMA), with fourth-generation (4G) wireless communicationprotocols, and/or the like. The mobile device 100 may also be configuredto operate in accordance with non-cellular communication mechanisms,such as via a wireless local area network (WLAN) or othercommunication/data networks.

The network interface 160 may also include an object recognitioninterface 173 in order to allow a user to execute some or all of theabove-described processes with respect to the object recognitionapplication 125 and/or the integrated circuit 180. The objectrecognition interface 173 may have access to the hardware, e.g., thetransceiver, and software previously described with respect to thenetwork interface 160. Furthermore, the object recognition interface 173may have the ability to connect to and communicate with an external datastorage on a separate system within the network as a means ofrecognizing the object(s) in the video stream.

As described above, the mobile device 100 may have a user interface thatincludes user output devices 136 and/or user input devices 140. The useroutput devices 136 may include a display 130 (e.g., a liquid crystaldisplay (LCD) or the like) and a speaker 132 or other audio device,which are operatively coupled to the processor 110. The user inputdevices 140, which may allow the mobile device 100 to receive data froma user 210, may include any of a number of devices allowing the mobiledevice 100 to receive data from a user, such as a keypad, keyboard,touch-screen, touchpad, microphone, mouse, joystick, other pointerdevice, button, soft key, and/or other input device(s).

The mobile device 100 may further include a power source 115. Generally,the power source 115 is a device that supplies electrical energy to anelectrical load. In one embodiment, power source 115 may convert a formof energy such as solar energy, chemical energy, mechanical energy, etc.to electrical energy. Generally, the power source 115 in a mobile device100 may be a battery, such as a lithium battery, a nickel-metal hydridebattery, or the like, that is used for powering various circuits, e.g.,the transceiver circuit, and other devices that are used to operate themobile device 100. Alternatively, the power source 115 may be a poweradapter that can connect a power supply from a power outlet to themobile device 100. In such embodiments, a power adapter may beclassified as a power source “in” the mobile device.

The mobile device 100 may also include a memory 120 operatively coupledto the processor 110. As used herein, memory may include any computerreadable medium configured to store data, code, or other information.The memory 120 may include volatile memory, such as volatile RandomAccess Memory (RAM) including a cache area for the temporary storage ofdata. The memory 120 may also include non-volatile memory, which can beembedded and/or may be removable. The non-volatile memory mayadditionally or alternatively include an electrically erasableprogrammable read-only memory (EEPROM), flash memory or the like.

The memory 120 may store any of a number of applications or programswhich comprise computer-executable instructions/code executed by theprocessor 110 to implement the functions of the mobile device 100described herein. For example, the memory 120 may include suchapplications as an object recognition application 125, an augmentedreality (AR) presentation application 121, a web browser application122, a Short Message Service (SMS) application 123, an electronic mail(i.e., email) application 124, etc.

Referring to FIG. 2, a block diagram illustrating an object recognitionexperience 200 in which a user 210 utilizes a mobile device 100 tocapture a video stream that includes an environment 250 is shown. Asdenoted earlier, the mobile device 100 may be any mobile communicationdevice. The mobile device 100 has the capability of capturing a videostream of the surrounding environment 250. The video capture may be byany means known in the art. In one particular embodiment, the mobiledevice 100 is a mobile telephone equipped with an image capture device170 (e.g., a camera) capable of video capture.

The environment 250 contains a number of objects 240. Some of suchobjects 240 may include a marker 230 identifiable to an objectrecognition application that is either executed on the mobile device 100or within the wireless network. A marker 230 may be any type of markerthat is a distinguishing feature that can be interpreted by the objectrecognition application to identify specific objects 220. For instance,an AR marker may be alpha-numeric characters, symbols, logos, shapes,ratio of size of one feature to another feature, a product identifyingcode such as a bar code, electromagnetic radiation such as radio waves(e.g., radio frequency identification (RFID)), architectural features,color, etc. In some embodiments, the marker 230 may be audio and themobile device 100 may be capable of utilizing audio recognition toidentify words or unique sounds broadcast. The marker 230 may be anysize, shape, etc. Indeed, in some embodiments, the marker 230 may bevery small relative to the object 220 such as the alpha-numericcharacters that identify the name or model of an object 220, whereas, inother embodiments, the marker 230 is the entire object 220 such as theunique shape, size, structure, etc.

In some embodiments, the marker 230 is not actually a physical markerlocated on or being broadcast by the object. For instance, the marker230 may be some type of identifiable feature that is an indication thatthe object 220 is nearby (e.g., a tag or placard near the object). Insome embodiments, the marker 230 for an object 220 may actually be themarker 230 for a different object 220. For example, the mobile device100 may recognize a particular building as being “Building A.” Datastored in the data storage 171 may indicate that “Building B” is locateddirectly to the east and next to “Building A.” Thus, markers 230 for anobject 220 that are not located on or being broadcast by the object 220are generally based on fixed facts about the object 220 (e.g., “BuildingB” is next to “Building A”). However, it is not a requirement that sucha marker 230 be such a fixed fact. The marker 230 may be anything thatenables the mobile device 100 to interpret to a desired confidence levelwhat the object is. For another example, the mobile device 100, objectrecognition application 125 and/or AR presentation application 121 maybe used to identify a particular person as a first character from apopular show, and thereafter utilize the information that the firstcharacter is nearby features of other characters to interpret that asecond character, a third character, etc. are nearby, whereas withoutthe identification of the first character, the features of the secondand third characters may not have been used to identify the second andthird characters. This example may also be applied to objects outside ofpeople.

The marker 230 may also be or include social network data, such as dataretrieved or communicated from the Internet, such as tweets, blog posts,social networking site posts, various types of messages and/or the like.In other embodiments, the marker 230 is provided in addition to socialnetwork data as mentioned above. For example, the mobile device 100 maycapture a video stream and/or one or more still shots including a largegathering of people. In this example, as above, one or more peopledressed as characters in costumes may be present at a specifiedlocation. The mobile device 100, object recognition application 125,and/or the AR presentation application 121 may identify several socialnetwork indicators, such as posts, blogs, tweets, messages, and/or thelike indicating the presence of one or more of the characters at thespecified location. In this way, the mobile device 100 and associatedapplications may communicate information regarding the social mediacommunications to the user and/or use the information regarding thesocial media communications in conjunction with other methods of objectrecognition. For example, the mobile device 100, object recognitionapplication 125, and/or the AR presentation application 121 performingrecognition of the characters at the specified location may confirm thatthe characters being identified are in fact the correct characters basedon the retrieved social media communications. This example may also beapplied objects outside of people.

In some embodiments, the mobile device and/or server access one or moreother servers, social media networks, applications and/or the like inorder to retrieve and/or search for information useful in performing anobject recognition. In some embodiments, the mobile device and/or serveraccesses another application by way of an application programminginterface or API. In this regard, the mobile device and/or server mayquickly search and/or retrieve information from the other programwithout requiring additional authentication steps or other gatewaysteps.

While FIG. 2 illustrates that the objects 220 with markers 230 onlyinclude a single marker 230, it will be appreciated that the object 220may have any number of markers 230 with each equally capable ofidentifying the object 220. Similarly, multiple markers 230 may beidentified by the mobile device 100 such that the combination of themarkers 230 may be utilized to identify the object 220. For example, themobile device 100 may utilize facial recognition markers 230 to identifya person and/or utilize a separate marker 230, such as the clothes theperson is wearing to confirm the identification to the desiredconfidence level that the person is in fact the person the mobile deviceidentified. For example, the facial recognition may identify a person asa famous athlete, and thereafter utilize the uniform the person iswearing to confirm that it is in fact the famous athlete.

In some embodiments, a marker 230 may be the location of the object 220.In such embodiments, the mobile device 100 may utilize GlobalPositioning System (GPS) hardware and/or software or some other locationdetermining mechanism to determine the location of the user 210 and/orobject 220. As noted above, a location-based marker 230 could beutilized in conjunction with other non-location-based markers 230identifiable and recognized by the mobile device 100 to identify theobject 220. However, in some embodiments, a location-based marker 230may be the only marker 230. For instance, in such embodiments, themobile device 100 may utilize GPS software to determine the location ofthe user 210 and a compass device or software to determine whatdirection the mobile device 100 is facing in order to identify theobject 220. In still further embodiments, the mobile device 100 does notutilize any GPS data in the identification. In such embodiments, markers230 utilized to identify the object 220 are not location-based.

FIG. 3 illustrates a mobile device 100, specifically the display 130,wherein the device 100 has executed an object recognition application125 and/or an AR presentation application 121 to present within thedisplay 130 indications of recognized objects within the vide stream(i.e., surrounding environment 250). The mobile device 100 is configuredto rely on markers 230 to identify objects 220 that are associated withrepresentations of objects, such as screenshots, depictions,advertisements, portions of, identifying marks, alphanumericidentifiers, and/or other representations of subjects, and indicate tothe user 210 the identified objects 220 by displaying an indicator 300on the mobile device display 130 in conjunction with the display of thevideo stream. As illustrated, if an object 220 does not have any markers230 (or at least enough markers 230 to yield object identification), theobject 220 will be displayed without an associated indicator 300.

The object recognition application 125 may use any type of means inorder to identify desired objects 220. For instance, the objectrecognition application 125 may utilize one or more pattern recognitionalgorithms to analyze objects in the environment 250 and compare withmarkers 230 in data storage 171 which may be contained within the mobiledevice 100 (such as within the integrated circuit 180) or externally ona separate system accessible via the connected network. For example, thepattern recognition algorithms may include decision trees, logisticregression, Bayes classifiers, support vector machines, kernelestimation, perceptrons, clustering algorithms, regression algorithms,categorical sequence labeling algorithms, real-valued sequence labelingalgorithms, parsing algorithms, general algorithms for predictingarbitrarily-structured labels such as Bayesian networks and Markovrandom fields, ensemble learning algorithms such as bootstrapaggregating, boosting, ensemble averaging, combinations thereof, and thelike.

Upon identifying an object 220 within the real-time video stream, the ARpresentation application 121 is configured to superimpose an indicator300 on the mobile device display 130. The indicator 300 is generally agraphical representation that highlights or outlines the object 220 andmay be activatable (i.e., include an embedded link) such that the user210 may “select” the indicator 300 and retrieve information related tothe identified object. The information may include any desiredinformation associated with the selected object and may range from basicinformation to greatly detailed information. In some embodiments, theindicator 300 may provide the user 210 with an internet hyperlink tofurther information on the object 220. The information may include, forexample, all types of media, such as text, images, clipart, video clips,movies, or any other type of information desired. In yet otherembodiments, the indicator 300 information related to the identifiedobject may be visualized by the user 210 without “selecting” theindicator 300.

In embodiments in which the indicator 300 provides an interactive tab tothe user 210, the user 210 may select the indicator 300 by anyconventional means, e.g., keystroke, touch, voice command, irisrecognition, or the like, for interaction with the mobile device 100.For instance, in some embodiments, the user 210 may utilize an inputdevice 140 such as a keyboard to highlight and select the indicator 300in order to retrieve the information. In a particular embodiment, themobile device display 130 includes a touch screen that the user mayemploy to select the indicator 300 utilizing the user's finger, astylus, or the like.

In some embodiments, the indicator 300 is not interactive and simplyprovides information to the user 210 by superimposing the indicator 300onto the display 130. For example, in some instances it may bebeneficial for the AR presentation application 121 to merely identify anobject 220, just identify the object's name/title, give briefinformation about the object, etc., rather than provide extensive detailthat requires interaction with the indicator 300. The AR presentationapplication 121 is capable of being tailored to a user's desiredpreferences.

Furthermore, the indicator 300 may be displayed at any size on themobile device display 130. The indicator 300 may be small enough that itis positioned on or next to the object 220 being identified such thatthe object 220 remains discernable behind the indicator 300.Additionally, the indicator 300 may be semi-transparent or an outline ofthe object 220 such that the object 220 remains discernable behind orenclosed by the indicator 300. In other embodiments, the indicator 300may be large enough to completely cover the object 220 portrayed on thedisplay 130. Indeed, in some embodiments, the indicator 300 may cover amajority or the entirety of the mobile device display 130.

The user 210 may opt to execute the object recognition application 125and/or the AR presentation application 121 at any desired moment andbegin video capture and analysis. However, in some embodiments, theobject recognition application 125 and/or the AR presentationapplication 121 includes an “always on” feature in which the mobiledevice 100 is continuously capturing video and analyzing the objects 220within the video stream. In such embodiments, the object recognitionapplication 125 may be configured to alert the user 210 that aparticular object 220 has been identified. The user 210 may set anynumber of user preferences to tailor the object recognition and ARexperience to their needs. For instance, the user 210 may opt to only bealerted if a certain particular object 220 is identified. Additionally,it will be appreciated that the “always on” feature in which video iscontinuously captured may consume the mobile device power source 115more quickly. Thus, in some embodiments, the “always on” feature maydisengage if a determined event occurs such as low power source 115, lowlevels of light for an extended period of time (e.g., such as if themobile device 100 is in a user's pocket obstructing a clear view of theenvironment 250 from the mobile device 100), if the mobile device 100remains stationary (thus receiving the same video stream) for anextended period of time, the user sets a certain time of day todisengage, etc. Conversely, if the “always on” feature is disengaged dueto the occurrence of such an event, the user 210 may opt for the “alwayson” feature to re-engage after the duration of the disengaging event(e.g., power source 115 is re-charged, light levels are increased,etc.).

In some embodiments, the user 210 may identify objects 220 that theobject recognition application 125 does not identify and add it to thedata storage 171 with desired information in order to be identifiedand/or displayed in the future. For instance, the user 210 may select anunidentified object 220 and enter a name/title and/or any other desiredinformation for the unidentified object 220. In such embodiments, theobject recognition application 125 may detect/record certain markers 230about the object so that the pattern recognition algorithm(s) (or otheridentification means) may detect the object 220 in the future.Furthermore, in cases where the object information is within the datastorage 171, but the object recognition application 125 fails toidentify the object 220 (e.g., one or more identifying characteristicsor markers 230 of the object has changed since it was added to the datastorage 171 or the marker 230 simply was not identified), the user 210may select the object 220 and associate it with an object 220 alreadystored in the data storage 171. In such cases, the object recognitionapplication 125 may be capable of updating the markers 230 for theobject 220 in order to identify the object in future video streams.

In addition, in some embodiments, the user 210 may opt to edit theinformation or add to the information provided by the indicator 300. Forinstance, the user 210 may opt to include user-specific informationabout a certain object 220 such that the information may be displayedupon a future identification of the object 220. Conversely, in someembodiments, the user may opt to delete or hide an object 220 from beingidentified and an indicator 300 associated therewith being displayed onthe mobile device display 130.

Furthermore, in some instances, an object 220 may include one or moremarkers 230 identified by the object recognition application 125 thatleads the object recognition application 125 to associate an object withmore than one object in the data storage 171. In such instances, theuser 210 may be presented with the multiple candidate identificationsand may opt to choose the appropriate identification or input adifferent identification. The multiple candidates may be presented tothe user 210 by any means. For instance, in one embodiment, thecandidates are presented to the user 210 as a list wherein the“strongest” candidate is listed first based on reliability of theidentification. Upon input by the user 210 identifying the object 220,the object recognition application 125 may “learn” from the input andstore additional markers 230 in order to avoid multiple identificationcandidates for the same object 220 in future identifications.

Additionally, the object recognition application 125 may utilize othermetrics for identification than identification algorithms. For instance,the object recognition application 125 may utilize the user's location,time of day, season, weather, speed of location changes (e.g., walkingversus traveling), “busyness” (e.g., how many objects are in motionversus stationary in the video stream), as well any number of otherconceivable factors in determining the identification of objects 220.Moreover, the user 210 may input preferences or other metrics for whichthe object recognition application 125 may utilize to narrow results ofidentified objects 220.

In some embodiments, the AR presentation application 121 may have theability to gather and report user interactions with displayed indicators300. The data elements gathered and reported may include, but are notlimited to, number of offer impressions; time spent “viewing” an offer,product, object or business; number of offers investigated via aselection; number of offers loaded to an electronic wallet and the like.Such user interactions may be reported to any type of entity desired. Inone particular embodiment, the user interactions may be reported to afinancial institution and the information reported may include customerfinancial behavior, purchase power/transaction history, and the like.

In various embodiments, information associated with or related to one ormore objects that is retrieved for presentation to a user via the mobiledevice may be permanently or semi-permanently associated with theobject. In other words, the object may be “tagged” with the information.In some embodiments, a location pointer is associated with an objectafter information is retrieved regarding the object. In this regard,subsequent mobile devices capturing the object for recognition mayretrieve the associated information, tags and/or pointers in order tomore quickly retrieve information regarding the object. In someembodiments, the mobile device provides the user an opportunity to postmessages, links to information or the like and associate such postingswith the object. Subsequent users may then be presenting such postingswhen their mobile devices capture and recognize an object. In someembodiments, the information gathered through the recognition andinformation retrieval process may be posted by the user in associationwith the object. Such tags and/or postings may be stored in apredetermined memory and/or database for ease of searching andretrieval.

Turning to FIG. 4, a high level overview of a method 400 of presentinginformation based on a representation of a subject is provided. In anembodiment, the method includes recognizing, via a mobile device, therepresentation of the subject; identifying, via a computing deviceprocessor, the subject based on an analysis of the representation;retrieving information corresponding to the subject; and presentinginformation using the mobile device, the information related to thesubject presented in conjunction with presentation of therepresentation.

Throughout this specification, the terms “AR system,” “AR device,” and“AR method” will be used. These terms refer to the aggregatefunctionality described herein. For example, the AR system can refer tothe mobile device 100 in conjunction with the computing device processor110, AR presentation application 121, Integrated circuit 180, networkedresources, and/or any data storage and input/output devices, asdisclosed herein. The AR method can refer to the use of all or part ofthe AR system to present information related to subjects based onidentified representations. In an embodiment, this presentation occursin an AR environment. The AR environment is not limited to real-time orconcurrent presentation of information. For example, the presentation ofinformation can be provided after a delay, such as by presenting anemailed list of all representations identified in a recorded videostream or saved still shot.

As used herein, objects are the visual or audible entities that arecaptured in a video or still shot. Objects include both subjects, whichare entities that can be captured in a video or still shot, andrepresentations, which are entities that correspond to subjects thatcannot easily be captured in a video or still shot. An example of asubject is an automobile while an example of a representation is atrademark on an automobile, wherein the trademark corresponds to thecompany manufacturing the automobile. The subject corresponding to arepresentation can also be capturable in a video or still shot but forsome reason the user is not able to do so when the user desiresinformation on the subject. For example, an individual can be easilycaptured in a video or still shot but the individual may not be present.In this case, the user of the AR system and method described herein canuse a photograph of the individual as a representation to receiveinformation about the individual depicted in the photograph.

Turning now to block 402, the computing device processor 110 or a remoteprocessor recognizes, via a mobile device, a representation of asubject. A representation is a broad concept and means anything thatcorresponds to a subject without actually being the subject. Therepresentation can range from a very high level abstraction to one thatis minimally distinguished from the subject. In one embodiment, forexample, the representation is a screenshot of the subject, such as ascreenshot of an application running on a mobile device, computer, orvideo game system. In another embodiment, the representation is adepiction of the subject, such as a photograph or painting depicting asubject. In some embodiments, the representation is an advertisement fora subject, such as a movie poster or newspaper advertisement. In anotherembodiment, the representation is a portion of the subject, such as arecorded portion of a television show or online video. In anotherembodiment, the representation is an identifying mark, such as atrademark, logo, barcode, or alphanumeric identifier. Each type ofrepresentation will be discussed in greater detail later with respect toFIGS. 6-10. Further, the types of representations disclosed herein arenot limiting. Other types of representations are possible and the ARsystems and methods disclosed herein are capable of recognizing them andproviding information related to the subjects corresponding to thoserepresentations.

In block 404, the computing device processor 110 or a remote processoridentifies the subject based on an analysis of the representation inconjunction with the AR presentation application 121. In someembodiments, the AR system and method concurrently provides informationrelated to the subject during depiction of the representation as anobject in the AR environment. For example, if a screenshot of anapplication is the representation, the subject is the application itselfand the information provided in the AR environment will be related tothe application. The AR presentation application 121 identifies thesubject based on an analysis of the representation. In an embodiment,the AR application analyzes markers on or associated with therepresentation and, based on those markers, identifies the subject. Insome embodiments, the analysis includes identifying one or more markersand then matching those markers with a listing of markers stored in oneor more databases and/or memory devices. The listing of markers, in someembodiments, includes data related to markers, their respectiveassociation(s) with representations and/or links to the subjectsassociated with those representations. In one example, diagnosticfeatures of screenshots for applications are stored in the database. TheAR system searches the AR environment for objects that depict thediagnostic features and then correlates the features with theapplication. In another example, the database includes markersassociated with trademarks and the companies represented by thosetrademarks. When the AR system and method identifies the markersassociated with the trademarks, the company associated with thattrademark is identified by retrieving the information from a database,such as data storage 171 on an Integrated circuit 180 or in the mobiledevice memory 120. In some embodiments, some or all the database is partof the mobile device and in other embodiments, some or all the databaseis located remotely to the mobile device. In some such embodiments, atleast a part of the database is stored separately from the mobile deviceand accessed over a network.

Turning now to block 406, the mobile device 100 retrieves informationcorresponding to the subject. In some embodiments, the mobile deviceretrieves the information from the mobile device memory 120 or datastorage 171 on the Integrated circuit 180. In another embodiment, themobile device retrieves the information over a network, such as the samenetwork used to identify the subject from the representation. In anotherembodiment, the mobile device retrieves information from a securenetwork associated with a financial institution. In another embodiment,the mobile device retrieves the information from a locally providednetwork. For example, a business may have a local wireless network thatincludes enhanced AR information for objects sold by the business.

The information received by the mobile device can vary broadly. In anembodiment, the information varies based on the subject, theenvironment, and/or the user. Types of information received will bediscussed in detail with regards to the specific examples in FIGS. 6-10.In some embodiments, the information is customized with the user's orthe subject's financial information. For example, the information on anapplication can provide the option to purchase the subject directly fromthe AR environment. The information can be customized with the user'sfinancial accounts so that the user can select the account that will beused to purchase the subject. In another example, the financialinformation is customized based on the environment. For example, if theAR system and method determines, via the AR presentation application121, that the user is present at a car dealership, e.g., the environmentincludes a number of cars of a single make identified by hood ornaments,the AR system and method can present information related to car loansthat the user may qualify for based on income, down payment, creditrating, and/or other financial information.

In block 408, the mobile device 100 presents the information, theinformation related to the subject presented in conjunction withpresentation of the representation. As discussed previously, in anembodiment the information is presented in real time concurrently withthe video or still shot, such as on the mobile device display 130. Insome embodiments, the information is presented as an overlay on top ofthe video or still shot. In other embodiments, the information ispresented by itself, that is, not in conjunction with the video or stillshot. In some embodiments, for example, a pop-up box can appear next toa screenshot of an application and allow the user to view the aggregateratings for the application. In an embodiment, the pop-up box isselectable and allows the user to open up a webpage or application thatprovides more detailed information than can be presented in the ARenvironment. The information can be presented temporarily or permanentlywhen the AR environment captures the representation. In anotherembodiment, the information is presented when the user moves a cursorover the representation, views the representation, or contacts thelocation on a touchscreen where the representation is depicted. Theinformation can be presented visually, such as in 2D or 3D, or audibly,such as a recording that plays when a representation is captured by theAR system, or both visually and audibly.

To more clearly describe the relationships between objects, markers,subjects, and representations, FIG. 5 provides a block diagramillustrating an AR experience 500 in which a user 310 utilizes a mobiledevice 312 to capture video or still shots of an environment 250. Theenvironment 250 includes objects 220 that are representations 342, 344corresponding to unseen subjects. In an embodiment, the unseen subjectsinclude both abstract subjects 343, such as trademarks, and unavailablesubjects 345, such as individuals. For example, the environment caninclude an automobile hood ornament that corresponds to an automobilecompany as well as a picture of an individual that corresponds to thatindividual. The automobile company is an abstract subject, i.e., it isvery difficult to capture an automobile company in a video or stillshot. The individual, however, is merely an unavailable subject. If theindividual were available, the individual could be captured in a videoor still shot. In some embodiments, the environment 250 also includessubjects 346 that can be captured in a real-time video, such as anautomobile. Both the representations 342, 344 and the subjects 346 thatcan be captured in a real-time video include markers (not shown) thatare used for identification purposes by the AR system and method. The ARsystem may also provide recognition by the sensing of different lightfrequencies (i.e. X-ray, UV, infrared, radio, etc.) in order to havevision through objects for further object detection.

Turning briefly to FIG. 6, an example of an AR environment includingboth representations and subjects is provided. The mobile device 312captures a second mobile device 602 in a video stream or screenshot 440.Application icons 608, 610 are depicted on the screen of the secondmobile device 602 and represent the applications themselves. The ARdevice and method can identify the application icons 608, 610 andprovide information related to the applications themselves through theAR environment. Additionally, the second mobile device 602 is a subjectthat can be captured directly. Information related to the second mobiledevice 602, such as the name of the device, the manufacturer of thedevice, or links to purchase the device can also be provided in the ARenvironment. This example is merely meant to illustrate that theenvironment 250 can include both representations 342, 344 and subjects346 that can be captured in video. Both are depicted as objects 220 inthe environment 250.

In FIG. 5, the mobile device 312 is capable of communicating over anetwork 502 with databases such as an AR database 504, a financialinstitution database 506, and/or any other type of database. In anembodiment, this allows the AR system to identify the representations342, 344 based on markers stored in the databases and then retrieveinformation related to the corresponding subjects 343, 345 from thedatabases. This information is then presented to the user 310 throughthe mobile device in conjunction with the display of the representations342, 344, as discussed previously.

In some embodiments, the information provided by the real-time videostream may be compared to data provided to the system through an API. Inthis way, the data may be stored in a separate API and be implemented byrequest from the mobile device and/or server accesses anotherapplication by way of an API.

Turning again to FIG. 6, an embodiment of the invention is disclosedwherein the representation is a screenshot of the subject on a computingdevice. For example, the screenshot can be of an application executingor installed on a computing device. In one embodiment, the applicationis on a mobile device, such as a portable phone or PDA. The applicationmay also be installed on or executing on a laptop, desktop, or tabletcomputer. In some embodiments, the application is a computer game on agaming system, television, or handheld gaming device. In one embodiment,the application is recognized by a screenshot of an icon placed on thedesktop or screen of the computing device after the application isinstalled. In another embodiment, the application is recognized by ascreenshot of the application during execution. FIG. 6 depicts a secondmobile device 602 having an application 606 executing on the screen 604while two icons 608, 610 representing additional applications arepresent at the base of the screen 604. In one embodiment, the AR systemand method described herein allows one user to take a real-time video ofan application using a first mobile device 312, wherein the applicationis executing on a second mobile device 602. The AR depiction 440 on thefirst mobile device 312 presents information relating to the applicationor applications 606, 608, 610 on the second mobile device 602. The ARdepiction 440 can present the information immediately or concurrentlywith the action taking place on the second mobile device 602, or the ARdepiction 440 can be called up at a later point and present theinformation then.

In one embodiment, the AR system and method presents information relatedto each of the applications through call-outs, pop-ups, or selectablelinks 612, 614, 616, 618. In another embodiment, the AR system andmethod presents information audibly when a representation is highlightedor selected. In a still further embodiment, the AR system and methodpresents information in a list or written form, such as in a document,webpage, email, or text message. Other types of presentation arepossible and the aforementioned list is not intended to limit the mannerby which the information relating the subjects can be presented.

The information presented relates to the applications themselves, and insome cases relates to specific characteristics of applications such asfrequent updating, frequently changing prices, customization, and theexistence of the same application across multiple computing platforms.In an embodiment, the information includes the name of the application612, the manufacturer of the application, the cost of the application,the version of the application, the operating systems that theapplication is capable of executing on, and other general informationrelated to the application. For example, the AR system can identify anapplication and determine whether the application is a more recentversion or upgrade of the application than the version existing on theuser's mobile device. Applications are frequently updated and being ableto identify new versions provides valuable information to users. Inanother embodiment, the information provides a link to furtherinformation about the application. For example, the AR system canidentify an application and then provide a link to reviews or rating forthe application 616. In one embodiment a general rating is provided andthen a link to detailed or individual user reviews is provided. In astill further embodiment, the category of the application can bedetermined and similar applications within the same category can bepresented to the user.

In a still further embodiment, information on the application caninclude a link to download 614 and/or install the application throughthe AR environment. In one embodiment, the information can be customizedfor the user of the system and/or method by including the user'sfinancial information with the application. The AR system canautomatically purchase the application when selected in the ARenvironment. In another embodiment, the application itself can becustomized with the user's financial information 618. For example, theuser's previous purchases at a business can be uploaded into anapplication related to the business. In some cases, this may provide theuser with enhanced privileges based on customer loyalty programs or thelike. Many finance and budgeting applications can be enhanced byfinancial information of the user, including account numbers,transactions, and/or balance information. In a still further embodiment,the financial information used to customize the information relates tothe cost history of the product. The financial institution may be ableto track the price of certain types of products or subjects depicted inthe AR environment and add value to the user's experience by updatingthe subject information with the price history of the subjectcorresponding to the representation.

Turning now to FIG. 7, an embodiment of the invention is disclosedwherein the representation is a depiction 700 of the subject orsubjects. In an embodiment, the depiction is a photograph of thesubject. For example, individuals 702, 704, 706 can be depicted in aphotograph and identified by markers specific to those individuals.Analysis can be performed on the photograph using facial recognitionsoftware or other analytical devices and information pertaining to theidentified individuals can be presented in the real-time AR environment.In another embodiment, the depiction is a picture or painting of thesubject. In some embodiments, the picture or painting accuratelyrepresents the subject so that the AR system identifies the subject. Inother embodiments, the markers alone are sufficient to identify subjectsfrom abstract pictures or paintings. For example, if a bowtie and a redhat are diagnostic features of an individual, the AR system and methodcan identify the individual from a painting of the individual where thebowtie and red hat are depicted. This identification is often assistedby contextual clues in the AR environment. For example, if theindividual is a representative of a business and a trademarkrepresenting the business is also present in the picture or painting,then the AR system can evaluate the individual using the additionalmarkers present in the environment.

The information presented in the AR environment with the representationscorresponds to the subjects depicted by the representations. When thesubject is an individual, the individual's name 708, contact information710, options to add the individual as a contact 712, links toinformation such as social networking sites related to the individual714, and the like, can be presented with the representation in the ARenvironment. Time sensitive information about individuals identified inthe AR environment can also be presented, such as links to buy ticketsto the individual's upcoming performances, reminders about theindividual's upcoming birthday 716, and other time-sensitiveinformation.

Depictions of media (not shown) can also be analyzed by the AR system.For example, the cover of a book, the cover of a CD, or the cover of amagazine can be analyzed based on a depiction in a photograph, picture,or painting. A photograph of a book cover can be analyzed by the ARsystem and markers associated with the title, the subjects depicted onthe cover, and any other diagnostic features on the cover can beidentified. Information such as the author of the book, other books bythe author, reviews of the book, a link to purchase the book, and thelike can be presented in the AR environment with the presentation of therepresentation of the book.

Information presented in conjunction with depictions can also becustomized with financial information. In an embodiment, the user canidentify the individual in the depiction and initiate a transfer ofmoney to or from the individual 718. In another embodiment, the ARenvironment can present financial information associated with thesubjects depicted in the AR environment. For example, prices of productsdepicted in the AR environment can be presented along with thedepiction. In some embodiments, the AR system includes privacy controlsto allow individuals represented in depictions to control theinformation presented in association with their depiction. If anindividual authorizes release of information, the AR environment can besupplemented with said information. For example, an individual mayauthorize release of information related to businesses patronized by theindividual. A merchant using an embodiment of the invention couldcapture a depiction of the individual and recognize that the individualhas made a purchase at her store.

In FIG. 8, an embodiment of the invention is disclosed wherein therepresentation is an advertisement 800 for a subject. The advertisementcan be a flyer, poster, or other type of visual advertisement for thesubject. In the example depicted in FIG. 8, a movie poster 802 iscaptured in a real-time video stream. The AR system and methodidentifies the movie represented by the poster advertisement andpresents information, such as reviews of the movie 804, options to buytickets to the movie 806, or a DVD of the movie 808, or profiles of theactors in the movie. In an embodiment, the AR system and method presentsthe information close to specific representations in the advertisement.For example, in FIG. 8, a biography 810 of the actor is presented closeto the actor's name in the movie poster. Additionally, furtherinformation related to the actor, such as other movies the actor was in812, can be presented in the AR environment 250. In another embodiment,the advertisement is in a newspaper or flyer (not shown). Theinformation presented in association with the advertisement may bedetails of a sale occurring at the business in the advertisement, or alink to the business's website. In a still further embodiment, theadvertisement is a movie preview and the AR system identifies the moviebeing advertised in the preview. In some embodiments, the advertisementis audible in nature. For example, the AR system may detect a commercialtheme song or product jingle in the AR environment and presentinformation relating to the product being advertised in the commercial.

The type of information that can be presented in association with anadvertisement includes information on the subject being advertised,options to interact with the subject, and associated information.Advertisements include many options for presenting informationcustomized by financial information from a financial institution. Forexample, information presented in conjunction of a subject may includethe sales price or the price history from all purchasers of the subjectavailable in the financial institution's database. For example, the ARenvironment can not only allow the user to purchase an item directly butcan also inform the user of the remaining balance in the user's accountafter the purchase. The user can include a budget associated with thefinancial accounts and the AR environment can present informationrelated to the user's budget with the advertisement, e.g., you budgeted$100 for entertainment this month and this show costs $40, leaving you$60 left in your entertainment budget.

In FIG. 9, an embodiment of the invention is disclosed wherein therepresentation is a portion of the subject 900 that is captured in avideo or screenshot. For example, a portion of a television show may berecorded and identified by the AR system and method. In another example,a portion of a movie, music video, text from within a book, onlinevideo, song, lecture, and the like are captured by the AR system andidentified. In the example depicted in FIG. 9, a real-time video of atelevision show 902 is captured by the AR system and method. Thetelevision show is identified by the portion of the television show thatis recorded and information relating to the show itself is presented tothe user. For example, the name of the show, the option to buy the DVDof the show 904, and in some embodiments the ability to set a homerecording system such as a DVR to record the show remotely 906 ispresented to the user. Further information, such as a plot summary ofthe specific episode 908 can also be provided once the subject, i.e.,the show itself, is identified. Online videos and movies can also becaptured and supplemented in a similar manner. In another embodiment,text from within a book or magazine is identified and information ispresented about the entire book. For example, a quote from a book may bereproduced on a sheet of paper. The AR system and method can capture thetext on the sheet of paper and identify the book from which the passagecomes. Links to purchasing the book, reviews of the book, the textbefore and after the representative text, and the like can be includedin the presentation. In a still further embodiment, a live lecture canbe recorded and supplemented with information presented related to thesubject of the lecture. For example, keywords, pictures, or the lecturercan be identified as described herein and the information can bepresented as the lecture is occurring. This can be advantageously usedin educational settings to present information related to the subjectmatter that is not discussed in the lecture.

The type of information that can be presented in conjunction with aportion of a subject includes the full title and identifying informationof the portion, a link to the full subject, or the like. The AR systemand method advantageously allows users to identify subjects from a mereportion of the subject captured in a mobile device. By providing thecapability to identify the full subject from a representative portion ofthe subject, the AR system and method described herein can supplementthe portion with detailed information. In some embodiments, theinformation related to the subject can be customized using financialinformation, as described herein.

Turning now to FIG. 10, an embodiment of the invention is disclosedwherein the representation is an identifying mark 1000. In someembodiments, the identifying mark is a trademark, service mark, logo,insignia, bar code, or other mark that identifies a product or producer.In another embodiment, the representation is an alphanumeric identifiersuch as an address, email address, phone number, URL address, orgeographic coordinates. In an embodiment, the AR system and methodrecognizes the identifying mark or alphanumeric identifier based onmarkers associated with the mark or identifier. The marks and associatedmarkers can be stored locally on the mobile device or stored in a remotedatabase. In one embodiment, the identifying marks are associated with aspecific business, team, or organization. In another embodiment, theidentifying marks are associated with a specific person or place. Forexample, a phone number can be captured in a video or still shot and theAR environment can identify the person associated with the phone number.If pictures of the person are available online or in a database, such asthe user's local database on the mobile device, the AR device canprovide the individual's picture in conjunction with the user's name,email address, or telephone number. Street addresses can be identifiedand a street view picture of the address can be presented as informationoverlaid on the street address. In the example depicted in FIG. 10, asports team logo 1002 is present on a t-shirt 1004. The AR system andmethod is able of identifying the logo 1002 from markings and presentinginformation related to the team that the logo represents.

In some embodiments, information such as contact information, name, orcategory is provided with the representation. For example, the AR methodmay provide information relating to the team's schedule 1006. The ARmethod may provide information relating to the team's tickets, as wellas the ability to purchase tickets directly 1008. The information can becustomized using the user's financial information. For example, the ARmethod can determine that a user is a supporter of a team based onpurchases at the team's facility, on the team's website, or otherpurchases that can be identified as related to the team. Once known,this information can be used to customize offers in the presentedinformation. For example, a fan of a team does not need information onwhat the team logo represents. Instead, the individual may be interestedin the team schedule, seeing tickets and merchandise individualsassociated with the user purchased, or purchasing team gear and hencethis information can be preferentially presented to the user in the ARenvironment.

Thus, methods, systems, computer programs and the like have beendisclosed that provide for using real-time video analysis, such as AR orthe like to assist the user of mobile devices with commerce activities.Through the use real-time vision object recognition objects, logos,artwork, products, locations and other features that can be recognizedin the real-time video stream can be matched to data associated withsuch to assist the user with commerce activity. The commerce activitymay include, but is not limited to; conducting a transaction, providinginformation about a product/service, providing rewards basedinformation, providing user-specific offers, or the like. In specificembodiments, the data that matched to the images in the real-time videostream is specific to financial institutions, such as customer financialbehavior history, customer purchase power/transaction history and thelike. In this regard, many of the embodiments herein disclosed leveragefinancial institution data, which is uniquely specific to financialinstitution, in providing information to mobile devices users inconnection with real-time video stream analysis.

The systems, methods, computer program products, etc. described herein,may be utilized or combined with any other suitable AR-relatedapplication. Non-limiting examples of other suitable AR-relatedapplications include those described in the following U.S. ProvisionalPatent Applications, the entirety of each of which is incorporatedherein by reference:

U.S. Provisional Ser. No. Filed On Title 61/450,213 Mar. 8, 2011Real-Time Video Image Analysis Applications for Commerce Activity61/478,409 Apr. 22, 2011 Presenting Offers on a Mobile CommunicationDevice 61/478,412 Apr. 22, 2011 Real-Time Video Analysis for RewardOffers 61/478,394 Apr. 22, 2011 Real-Time Video Image Analysis forProviding Targeted Offers 61/478,399 Apr. 22, 2011 Real-Time AnalysisInvolving Real Estate Listings 61/478,402 Apr. 22, 2011 Real-Time VideoImage Analysis for an Appropriate Payment Account 61/478,405 Apr. 22,2011 Presenting Investment-Related Information on a Mobile CommunicationDevice 61/478,393 Apr. 22, 2011 Real-Time Image Analysis for MedicalSavings Plans 61/478,397 Apr. 22, 2011 Providing Data Associated WithRelationships Between Individuals and Images 61/478,408 Apr. 22, 2011Identifying Predetermined Objects in a Video Stream Captured by a MobileDevice 61/478,400 Apr. 22, 2011 Real-Time Image Analysis for ProvidingHealth Related Information 61/478,411 Apr. 22, 2011 Retrieving ProductInformation From Embedded Sensors Via Mobile Device Video Analysis61/478,403 Apr. 22, 2011 Providing Social Impact Information AssociatedWith Identified Products or Businesses 61/478,415 Apr. 22, 2011Providing Location Identification of Associated Individuals Based onIdentifying the Individuals in Conjunction With a Live Video Stream61/478,419 Apr. 22, 2011 Vehicle Recognition 61/478,417 Apr. 22, 2011Collective Network of Augmented Reality Users 61/508,985 Jul. 18, 2011Providing Information Regarding Medical Conditions 61/508,946 Jul. 18,2011 Dynamically Identifying Individuals From a Captured Image61/508,980 Jul. 18, 2011 Providing Affinity Program Information61/508,821 Jul. 18, 2011 Providing Information Regarding SportsMovements 61/508,850 Jul. 18, 2011 Assessing EnvironmentalCharacteristics in a Video Stream Captured by a Mobile Device 61/508,966Jul. 18, 2011 Real-Time Video Image Analysis for Providing VirtualLandscaping 61/508,969 Jul. 18, 2011 Real-Time Video Image Analysis forProviding Virtual Interior Design 61/508,971 Jul. 18, 2011 Real-TimeVideo Image Analysis for Providing Deepening Customer Value 61/508,764Jul. 18, 2011 Conducting Financial Transactions Based on Identificationof Individuals in an Augmented Reality Environment 61/508,973 Jul. 18,2011 Real-Time Video Image Analysis for Providing Security 61/508,976Jul. 18, 2011 Providing Retail Shopping Assistance 61/508,944 Jul. 18,2011 Recognizing Financial Document Images

While the foregoing disclosure discusses illustrative embodiments, itshould be noted that various changes and modifications could be madeherein without departing from the scope of the described aspects and/orembodiments as defined by the appended claims. Furthermore, althoughelements of the described aspects and/or embodiments may be described orclaimed in the singular, the plural is contemplated unless limitation tothe singular is explicitly stated. Additionally, all or a portion of anyembodiment may be utilized with all or a portion of any otherembodiment, unless stated otherwise.

While certain exemplary embodiments have been described and shown in theaccompanying drawings, it is to be understood that such embodiments aremerely illustrative of and not restrictive on the broad invention, andthat this invention not be limited to the specific constructions andarrangements shown and described, since various other changes,combinations, omissions, modifications and substitutions, in addition tothose set forth in the above paragraphs, are possible. Those skilled inthe art will appreciate that various adaptations and modifications ofthe just described embodiments can be configured without departing fromthe scope and spirit of the invention as disclosed herein. Therefore, itis to be understood that, within the scope of the appended claims, theinvention may be practiced other than as specifically described herein.

What is claimed is:
 1. A method for presenting information based on acaptured representation of a subject, the method comprising: determiningthat a mobile phone is capturing a real-time video of a scene, whereinthe scene comprises at least one representation of an individual ofinterest to a user of the mobile phone, wherein the representation is apicture of the individual; determining positioning information of themobile phone, wherein the positioning information is associated with thereal-time video being captured by the mobile phone; recognizing, via themobile phone, the representation of the individual in the real-timevideo based on an analysis of markers associated with therepresentation; identifying, via a computing device processor, theindividual using a database comprising a plurality of individuals andassociated representations; confirming the identification of theindividual to the desired confidence level; retrieving informationcorresponding to previous transfers of money between the user and theindividual; presenting the information using the mobile phone, theinformation related to the individual presented in conjunction withpresentation of the representation; and superimposing the informationover the video being captured by the mobile phone.
 2. The method ofclaim 1, further comprising initiating a transfer of money to theindividual via the mobile phone.
 3. The method of claim 1, wherein therepresentation is a depiction of the individual in a medium selectedfrom the group consisting of a photograph, a painting, a poster, and apicture.
 4. The method of claim 1, wherein the representation comprisesan identifying mark of the individual.
 5. An apparatus for presentinginformation based on a captured representation of a subject, theapparatus comprising: a computing platform having a processor, a memoryin communication with the processor, logic stored in the memory,executable by the processor and configured to determine that a mobilephone is capturing a real-time video of a scene, wherein the scenecomprises at least one representation of an individual of interest to auser of the mobile phone, wherein the representation is a picture of theindividual; positioning logic stored in the memory, executable by theprocessor and configured to determine positioning information of themobile phone, wherein the positioning information is associated with thereal-time video being captured by the mobile phone, image capture logicstored in the memory, executable by the processor and configured tocapture the video comprising a representation of the individual in thereal-time video based on an analysis of markers associated with therepresentation, analysis logic stored in the memory, executable by theprocessor and configured to analyze the representation in the video,image identification logic stored in the memory, executable by theprocessor and configured to identify the individual using a databasecomprising a plurality of individuals and associated representations,confirmation logic stored in the memory, executable by the processor andconfigured to confirm the identification of the individual to thedesired confidence level; retrieval logic stored in the memory,executable by the processor and configured to retrieve informationcorresponding to previous transfers of money between the user and theindividual; and presentation logic stored in the memory, executable bythe processor and configured to present the information related to theindividual in the video, and superimpose the information over the videobeing captured by the mobile phone.
 6. The apparatus of claim 5, furthercomprising a network interface device, wherein the network interfacedevice is configured to wirelessly connect to the database comprisingthe plurality of individuals and associated representations.
 7. Theapparatus of claim 5, further comprising customization logic stored inthe memory, executable by the processor, and configured to customize apresentation for a user.
 8. The apparatus of claim 7, further comprisingtransfer logic stored in the memory, executable by the processor andconfigured to initiate a transfer of money to the individual via themobile phone.
 9. The apparatus of claim 5, further comprising a display,wherein the information is presented on the display.
 10. A computerprogram product, comprising a non-transitory computer-readable mediumcomprising computer-executable instructions for presenting informationbased on a captured representation of a subject, the instructionscomprising: instructions for determining that a mobile phone iscapturing a real-time video of a scene, wherein the scene comprises atleast one representation of an individual of interest to a user of themobile phone, wherein the representation is a picture of the individual;instructions for determining positioning information of the mobilephone, wherein the positioning information is associated with thereal-time video being captured by the mobile phone; instructions forcapturing the video comprising a representation of the individual in thereal-time video based on an analysis of markers associated with therepresentation; instructions for analyzing the representation in thevideo; instructions for identifying the individual using a databasecomprising a plurality of individuals and associated representations;instructions for confirming the identification of the individual to thedesired confidence level; instructions for retrieving informationcorresponding to previous transfers of money between the user and theindividual; instructions for presenting the information related to theindividual in the video; and instructions for superimposing theinformation over the video being captured by the mobile device.
 11. Thecomputer program product of claim 10, further comprising instructionsfor initiating a transfer of money to the individual via the mobilephone.